1. Field of the Invention
The present invention relates to a technique of, in an image processing apparatus, correcting a distortion of a captured image caused by a shake of an image capturing apparatus using a method of electronically deforming an image.
2. Description of the Related Art
In recent years, CMOS image pickup devices are rapidly proliferating as image pickup devices used in image capturing apparatuses. When capturing a moving image using a CMOS image pickup device, control is performed to read charges accumulated on the line basis from the upper portion to the lower portion of the CMOS image pickup device. This read method is called a rolling shutter method in which the read timing changes between the upper portion and the lower portion of the image pickup device. If a shake occurs in the image capturing apparatus, and the object position moves on the imaging plane, a distortion (rolling shutter distortion) occurs in the captured image due to the difference of the charge read timing in the image pickup device.
As a method of correcting such a rolling shutter distortion, there is proposed a method of discretely acquiring the amounts of shakes that occur in the image capturing apparatus so as to synchronize with the read timings of the CMOS image pickup device and correcting the rolling shutter distortion based on the acquired shake amounts (for example, Japanese Patent Laid-Open No. 2007-264074).
In the above-described related art, however, the reference line of rolling shutter distortion correction (a line where the read position does not change in rolling shutter distortion correction) is not clear, and the following problem sometimes arises.
FIG. 14A is a view for explaining a method of correcting a rolling shutter distortion that has occurred in an object due to a shake applied to an image capturing apparatus. The range of the outermost rectangle in FIG. 14A indicates the range of the entire captured image of the image capturing apparatus. The large dotted rectangle in the captured image indicates the image read range without a rolling shutter distortion. The small dotted rectangle indicates the shape of the object without a rolling shutter distortion. The small rhombic pattern in FIG. 14A indicates a state in which a rolling shutter distortion occurs due to application of a shake to the image capturing apparatus, and the captured object shrinks in the vertical direction and distorts obliquely.
Rolling shutter distortion correction is performed by changing the image read range in accordance with the rolling shutter distortion, as indicated by the large rhombic pattern in FIG. 14A. At this time, the reference line of rolling shutter distortion correction is set to the uppermost line of the captured image in FIG. 14A. In FIG. 14A, on the lines under the reference line, the read position is changed rightward relative to the reference line, thereby performing correction to return the image that has obliquely distorted to the undistorted state. Additionally, on the lines under the reference line, the read line position is gradually shifted upward, thereby performing correction to enlarge the image that has shrunk in the vertical direction and return it to the original size. The object that has distorted to a rhombus in FIG. 14A can thus be returned to the original shape (in this case, a rectangle).
FIG. 14B is a view showing an output obtained by performing the above-described rolling shutter distortion correction. The range of the outermost rectangle in FIG. 14B indicates the range of the output image after the rolling shutter distortion correction. The solid square pattern in the output image indicates the object obtained by performing the rolling shutter distortion correction for the object distorted into a rhombus in FIG. 14A. The dotted square pattern in FIG. 14B indicates the object on the output image when the image read range (large dotted rectangle in FIG. 14A) without a rolling shutter distortion in FIG. 14A is read.
In FIG. 14A, assume that the center position of the object distorted into a rhombus due to the rolling shutter distortion and the center position of the square object without a rolling shutter distortion are the same on the captured image. This assumption is placed for the descriptive convenience because, actually, a shake of the image capturing apparatus makes the object position vary on the captured image. At this time, when the rolling shutter distortion correction is performed while setting the reference line to the position shown in FIG. 14A, the object position shifts on the output image, as indicated by the solid and dotted squares in FIG. 14B. That is, without correct setting of the reference line of rolling shutter distortion correction, the object position shifts to result in a new blur even if the distorted object can be returned to the original shape.